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Motivation  and  Goals 


Our  goal  is  to  construct  numerical  methods  for  non-hydrostatic  mesoscale  and  global 
atmospheric  models  (for  NWP  applications);  this  is  a  unified  model.  The  reason  for  this 
is  economics  -  one  (production)  model  is  cheaper  to  support  or  at  the  very  least,  having 
the  same  dynamics  simplifies  a  great  many  things  (e.g.  training  future  developers). 

Our  aim  is  to  build  a  modeling  framework  with  the  following  capabilities: 

1 .  Highly  scalable  on  current  and  future  computer  architectures  (exascale  computing 
and  beyond  and  GPUs) 

2.  Flexibility  to  use  a  wide  range  of  grids  (e.g.,  statically  and  dynamically  adaptive) 

3.  Global  model  that  is  valid  at  the  meso-scale  (i.e.,  non-hydrostatic) 

4.  NUMA  Framework  to  include: 

-  A  suite  of  time-integrators  (explicit,  semi-implicit,  fully-implicit) 

-  Various  numerical  methods  (SE/DG) 

-  Various  forms  of  the  Governing  Equations  to  determine  what  form  is  more  accurate,  efficient,  robust. 


Talk  Summary 


1 .  How  do  computer  architectures  affect  the  models? 

2.  Numerical  Methods  in  New  Dynamical  Cores 

3.  How  does  resolution  affect  the  model  equations? 

4.  What  Should  we  aim  for  in  our  New  Models 

5.  Where  We  Plan  to  Head  with  NUMA 


Talk  Summary 


1 .  How  do  computer  architectures  affect  the  models? 

•  From  Terascale  to  Petascale/Exascale  Computing 

•  10  of  Top  500  are  already  in  the  Petascale  range 

•  Should  also  keep  our  eyes  on  GPUs  (e.g.,  Mare  Nostrum) 

2.  Numerical  Methods  in  New  Dynamical  Cores 

3.  How  does  resolution  affect  the  model  equations? 

4.  What  Should  we  aim  for  in  our  New  Models 

5.  Where  We  Plan  to  Head  with  NUMA 


Performance  of  a  Global 

Hydrostatic  Model 
(T239  L42) 
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30  day  simulation  for  a  3D  global  (hydrostatic)  atmospheric  model  for  a  Baroclinic 
Instability  (Giraldo  QJRMS  2005). 
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Performance  of  a  Global/Mesoscale 
Non-Hydrostatic  Model 
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Talk  Summary 


1 .  How  do  computer  architectures  affect  the  models? 

2.  Numerical  Methods  in  New  Dynamical  Cores 

•  Time-Integration  is  important  (e.g.,  explict,  semi-implicit,  hilly-implicit) 

•  Spatial  Discretization  methods  is  how  we  are  able  to  take  advantage  of 
Parallel  computers  (i.e.,  domain  decomposition  of  the  physical  grid) 

•  What  other  properties  should  we  strive  for  in  our  Numerical  Methods? 

3.  How  does  resolution  affect  the  model  equations? 

4.  What  Should  we  aim  for  in  our  New  Models 

5.  Where  We  Plan  to  Head  with  NUMA 


Summary  of  Numerical  Methods 


•  Proxy  for  Primitive  Equations: 


l=S(?)  — » %).| 


•  S(q)  must  be  approximated  discretely: 
Options  are: 

1.  Finite  Difference  Methods  (Taylor  Series) 


<7,+i  =  4,  +  Ax 
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2.  Galerkin  Methods  (Basis  Function 
Expansions) 
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3.  Godunov  Methods  (Control  Volume 
Approach) 

dt 


f  V^cKle  +  l  ¥(n-  FN  )dTe  -  J  V  y  •  Fw  =  0 


+  V-F  =  0 


L  v 


fdq  ^ 


IN 


V 


+  V  •  F; 


JQ  =0 


7 


at 


Spectral  Elements  in  a  Nutshell 


Spectral  Elements  in  a  Nutshell 


With  New  Models  Come  New  Grids 

(conduit  between  dynamics  and  physics) 
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Talk  Summary 


1 .  How  do  computer  architectures  affect  the  models? 

2.  Numerical  Methods  in  New  Dynamical  Cores 

3.  How  does  resolution  affect  the  model  equations? 

•  Hydrostatic  versus  Non-hydro  static  models  and  regimes  of  validity  of  the 

governing  equations 

4.  What  Should  we  aim  for  in  our  New  Models 

5.  Where  We  Plan  to  Head  with  NUMA 


Unified  Equations  Framework 
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When  are  these  equations  valid? 


1 .  Non-hydrostatic  Equations 

•  These  equations  are  valid  at  all  spatial  scales;  these  are  the  most  general  form  of 
the  governing  equations. 

•  The  problem  with  this  form  is  that  they  permit  acoustic  waves  which  are  very 
fast  and  have  little  effect  on  the  dynamical  processes  we  are  interested  in. 

•  We  know  how  to  solve  these  equations  well  but  requires  sophisticated  numerical 
methods  (DG,  semi-implicit,  etc.) 

2.  Hydrostatic  Equations 

•  These  are  very  simple  equations  to  solve  but  are  not  valid  if  you  are  interested  in 
vertical  acceleration.  Because  vertical  acceleration  is  omitted,  it  then  means  that 
the  vertically  propagating  acoustic  waves  are  removed  from  the  equations. 

•  This  approximation  is  no  longer  valid  below  10km  resolution.  Many  NWP 
models  are  either  at  this  limit  or  approaching  it  quickly  (e.g.,  IFS,  UM,  Grapes). 

3.  Pseudo-Incompressible  Equations 

•  It  would  be  nice  to  keep  the  equations  valid  below  10km  without  having  to  deal 
with  the  vertically  propagating  acoustic  waves;  these  equations  are  a  likely 
candidate. 

•  No  operational  NWP  or  climate  models  currently  use  this  approach,  only 
research  codes  (e.g.,  EULAG).  It  would  be  interesting  to  explore  this  form  of  the 
equations. 


Talk  Summary 


1 .  How  do  computer  architectures  affect  the  models? 

2.  Numerical  Methods  in  New  Dynamical  Cores 

3.  How  does  resolution  affect  the  model  equations? 

4.  What  Should  we  aim  for  in  our  New  Models 
E.g.,  Conservation,  Scalability,  High-order  Accuracy,  Adaptivity 

Where  We  Plan  to  Head  with  NUMA 


5. 


4.  What  Should  We  Aim  For? 


1 .  Conservation  -  Conservation  of  Mass  and  Energy  are  absolute  musts; 
what  else  should  we  conserve? 

2.  Scalability  -  New  models  must  be  highly  scalable  because  we  will 
continue  to  get  more  processors 

3.  High-Order  Accuracy  -  Accuracy  is  important,  of  course,  but  how  do  we 
measure  this  and  what  order  accuracy  is  sufficient?  This  question  is 
coupled  to  the  accuracy  of  the  physics,  data  assimilation,  etc.  From  the 
standpoint  of  scalability,  high-order  is  good  (hp  methods  =  on-processor 
work  is  large  but  the  communication  footprint  is  small).  This  is  also  a 
good  strategy  for  exploiting  MPI/Open  MP  Hybrid. 

4.  Adaptivity  -  Adaptive  methods  have  improved  tremendously  in  the  past 
decade  and  it  may  offer  an  opportunity  to  solve  problems  not  feasible  a 
decade  ago  but  we  need  to  identify  these  applications  (e.g.,  hurricanes, 
storm-surge  modeling). 


Some  Standing  Issues  for  Adaptive  Methods 


•  Parallelization/Domain  Decomposition:  Modifying  the  data  structures 
dynamically  slows  the  computations.  E.g.,  the  domain  decomposition  needs  to  be 
a  direct  by-product  of  the  adaptive  mesh  generator.  A  good  first  candidate  for  AM 
is  statically  adaptive  grids  where  the  grid  is  modified  and  held  fixed  for  the  entire 
simulation.  This  must  work  well  before  moving  onto  dynamically  adaptive  grids. 

•  Coupling  of  Dynamics  and  Physics:  Sub-grid  scale  parameterization  is 
notoriously  inconsistent  meaning  that  changing  the  grid  resolution  changes  the 
results.  Also,  the  dynamics  must  use  the  “proper”  approximations  for  the  smallest 
scales.  This  means  that  both  Atmospheric  and  Ocean  codes  should  use  the 
nonhydrostatic  equations.  This  has  direct  effects  on  the  time-integration  strategies 
as  faster  waves  (e.g.,  vertically  propagating  “acoustic”  waves)  must  be  considered 
in  the  choice  of  time-step. 


Non-hydrostatic  Adaptivity  Examples 

(Muller,  Behrens,  Giraldo,  Wirth  2010) 
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Warm  air  bubble 
after  0  seconds. 

amplitude:  0.5  K 
order  of  polynomials:  3 
timestep  length:  5ms 
shortest  edge  length:  11m 
diffusion  parameter:  0.1m2/s 
time  integration  method:  RK35 
color:  pot.  temperature  -  300K 


(c)  August  2009,  A.  Mueller  (Mainz,  Germany) 


Rising  Thermal  Bubbles 


Two  (Warm/Cold)  Thermal  Bubbles 


Talk  Summary 


1 .  How  do  computer  architectures  affect  the  models? 

2.  Numerical  Methods  in  New  Dynamical  Cores 

3.  How  does  resolution  affect  the  model  equations? 

4.  What  Should  we  aim  for  in  our  New  Models 

5.  Where  We  Plan  to  Head  with  NUMA 

•  History  of  NUMA 

•  Tested  in  Limited- Area  Mode  (Mesoscale  applications) 

•  Currently  testing  in  Global  Mode 

•  Unstructured  Grid  capability  should  facilitate  coupling  to  Unstructured  Ocean 
(and  Coastal  Ocean)  Models 


History  of  NUMA  Model 


•  Early  2000s:  Navy’s  Spectral  Element  Atmospheric  Model  (NSEAM) 

-  Similar  to  SEAM/HOMME  (see  Mark  Taylor’s  talk) 

-  Hydrostatic  Equations 

-  Based  on  Spectral  Elements 

-  Based  on  cubed-sphere  grid 

-  Tested  with  NOGAPS  physical  parameterization 

•  Late  2000s:  Prototype  2D  (x-z  slice)  Mesoscale  Atmospheric  Model 

-  Nonhydrostatic  Equations 

-  Based  on  Spectral  Elements  and  Discontinuous  Galerkin  methods 

-  Tested  with  simple  moist  physics  (Kessler  Physics) 


•  2010:  3D  Mesoscale  Atmospheric  Model  (NUMA) 

-  Nonhydrostatic  Equations 

-  Based  on  Spectral  Element  and  Discontinuous  Galerkin  methods 

-  Tested  as  Limited- Area  model  (NRBCs) 

-  Currently  being  tested  on  the  Sphere  as  a  Global  Model 

-  Moist  Physics  will  be  included  in  the  near  future 


Example  of  3D  Grids 


Mesoscale  Modeling  Mode  Global  Modeling  Mode  Global  Modeling  Mode 

(Cubed-Sphere)  (Icosahedral) 


NUMA  runs  in  either  Limited- Area  or  Global  Mode. 
Currently,  any  grid  can  be  used  including  completely 
unstructured  grids. 

Parallel  Domain  Decomposition  handled  by  METIS. 


Limited-Area  Mode: 


Linear  Hydrostatic  Isolated  Mountain 


•  Flow  ofU=20  m/s  in  an  isothermal  atmosphere. 

•  LH  Mountain:  Solid  of  revolution  of  Witch  of  Agnesi:  Mountain  height  =  1  m  with 
radius  10  km. 

•  Absorbing  (sponge)  boundary  condition  implemented  on  lateral  (4  sides)  and  top 
boundaries. 


LH  Isolated  Mountain 
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Global  Mode: 
Rising  Thermal  Bubble 


Summary 


•  The  Spectral  Element  method  allows  for  the  construction  of  accurate, 
efficient,  and  flexible  atmospheric  (and  possibly  ocean)  models. 

•  The  scalability  of  this  method  has  been  shown  previously  in  numerous  other 
applications  (see  Gordon  Bell  Prizes  awarded  in  CFD,  Seismic  Wave 
propagation,  etc.). 

•  The  geometric  flexibility  of  SE  facilitates  the  use  of  unstructured/adaptive 
grids  (e.g.,  for  tracking  hurricanes) 

•  SE  models  have  been  shown  to  work  well  for  the  Hydrostatic  Equations 
including  Physical  Parameterizations  (Kim  et  al.  JGR  2009,  for  MJO 
studies,  Baer  et  al.  MWR  2006,  Mark  Taylor’s  presentation) 

•  Although  high-order,  can  be  shown  to  be  fully  conserving  and  even  positive- 
definite  (recent  work  by  Mark  Taylor  and  Aime’  Fournier  JCP  2010). 

•  SE  models  have  been  shown  to  work  well  for  the  Non-hydrostatic  Equations 
(Giraldo-Restelli  JCP  2008,  Restelli-Giraldo  SISC  2009,  Giraldo  et  al.  SISC 
2010) 

•  SE  models  have  been  shown  to  work  well  with  simple  physical 
parameterizations  (Gabersek  et  al.  MWR  2010) 

•  Collaborations  between  DOE  (Sandia  via  Mark)  and  Navy  (NPS/NRL)  have 
already  occurred  but  should  strengthen  these  and  extended  to  include 
various  partners  (e.g.,  numerics,  physical  parameterization,  data 
assimilation,  new  grids,  new  applications) 


